Virtualization is one way to decouple software applications from an underlying hardware. The most common way of virtualization is to run a hypervisor either directly on an underlying hardware or on top of a host operating system (OS). A hypervisor emulates the hardware, allowing many virtual machines (VMs) to run on the same hardware. Each such VM is isolated from said other VMs and runs a full OS. Different VMs on the same hardware can run different OSs.
In contrast to hypervisors, containers are light weight and have much lower overhead. In container-based virtualization, also known as operating system level virtualization, containers share one and the same kernel, rather than running different OSs.
In Linux containers, the kernel-level namespaces are used for resource isolation and control groups (cgroups) are used for managing and limiting resources. Cgroups also expose resource usage metrics such as memory, central processing unit (CPU), and block input/output (I/O), which can be used for monitoring.
Container-based virtualization provides better resource utilization as compared to VMs since idle containers do not use any resources, whereas VMs do. Moreover, containers can be created and destroyed much faster as compared to VMs, since there is no need to boot and shutdown a full OS.
Some examples of container-based virtualization technologies in Linux are OpenVZ, Linux VServer, LXC, and Docker. More recent container virtualization projects include LXD and Rocket.
Communication between containers can take place using different methods. For example, Docker by default creates a virtual Ethernet bridge inside the kernel on the host machine. Data traffic on this virtual Ethernet bridge can be forwarded to every interface that is connected to it. When a new container is created, a pair of peer interfaces may be created. One of the peers becomes the interface for the container and the other ones can be bound to the default Docker Bridge or any other virtual bridge on the host machine. Containers running on different hosts, or server nodes, can communicate with each other via a virtual switch.
Existing tools for monitoring containers can only gather metrics about the containers themselves running on a host machine such as CPU, memory, block I/O usage. These metrics can be obtained from cgroups. Network metrics that can be collected per interface are limited to the number of packets and bytes received/transmitted from an interface.
There are a number of tools for monitoring Linux containers, particularly Docker containers, but none of these tools provide latency monitoring. Docker version 1.5 [Ref. 1] has introduced a statistics application programming interface (API) that can provide access to a live stream of CPU, memory, network I/O and block I/O for running containers.
Another example is CAdvisor [Ref. 2] that can create a monitoring container to monitor resource usage of running containers on a host. A further example is sFlow that has added support for Docker containers to obtain standard sFlow performance metrics for Linux containers [Ref. 3].
Due to emerging latency-sensitive applications, there is an increasing need for latency monitoring. Latency monitoring can be performed using two basically different measurement methods. One can be considered to be an active measurement method, and the other a passive measurement method.
Active measurement methods are based on injecting probe packets into network traffic and monitoring these injected probe packets. In passive measurement, in contrast, network traffic is being observed without injecting probe packets. A common assumption in passive latency measurement methods, is that the communicating sender and receiver are tightly synchronized in time.
A variety of different passive latency measurement methods have recently been proposed. LDA [Ref. 4] can provide aggregate latency measurements across all packets passing through two observation points. FineComb [Ref. 5] can provide end-to-end aggregate loss and latency measurements between edge routers by taking care of packet re-orderings.
Flow-aggregated latency measurements cannot resolve latency experienced by different flows. For the purpose of resolving latencies in different flows, per-flow latency measurement methods can be used. RLI [Ref. 6] can obtain per-flow measurements. However, this technique requires injecting reference packets into the data traffic.
MAPLE [Ref. 7] presents an architecture for latency monitoring where the granularity of measurements can be selected. However, it is therein assumed that network packets can carry timestamps.
Finally, COLATE [Ref. 8] provides a per-flow latency monitoring scheme which does not require the use of any probes or timestamps. However, this scheme consists of a recording phase and a querying phase and is not designed for real-time latency measurements.
As mentioned above, existing container monitoring platforms, such as cAdvisor, are limited to the monitoring of fairly simple network properties such as counting packets and bytes in and out of a container. It is mentioned that data packet delay and jitter are examples of metrics, which are not properly supported by the current monitoring approaches, for the reasons as described below.
Monitoring per-flow metrics using Host sFlow requires executing a daemon in the host that executes the containers and enabling sFlow support on the open virtual switch (OVS) infrastructure that interconnects the containers. It is a drawback that sFlow does not allow evaluating delay and jitter, and packet loss evaluation is local to the port being observed, which means that no end-to-end measurement is made possible.
Moreover, provisioning sFlow in an OVS instance connected to the container requires the container orchestration framework to have information about which switch is connected to which container, where this information could be local to the orchestration or contained in a network controller, and then communicate this information with the sFlow daemon. This is usually a management task that is performed outside of the control plane, largely in a manual or automated-through-scripts manner. This way of operation increases the reaction times to both unforeseen problems and normal events, such as migration, which may affect the lifecycle of a container.
Monitoring network activity of containers for virtualization is known and allows mainly for counting bytes and packets communicated to or from a container. More complex monitoring, for example, for observing per-flow packets and bytes, or calculating metrics such as packet loss, delay and jitter, requires cooperation from virtual or physical switches through which container communicates. Achieving such cooperation involves a complex interaction between cloud and network orchestration and provisioning platforms. Apart from the complexity of the interaction, this would also be inefficient in terms of reaching short, for example sub-second, reaction times to problems or regular changes such as container migrations or re-instantiations on different hardware.
There is hence a need for a solution addressing one or more of the issues as discussed above.